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Introduction. Members of the millennial generation (born after 1982) have come of age in a 
society infused with technology and information. It is unclear how they determine the validity of 
information gathered, or whether or not validity is even a concern. Previous information search 
models based on mediated searches with different age groups may not adequately describe the 
search behaviours of this generation. 

Method. The longitudinal study discussed here examined the information behaviour of 
undergraduate college students who were members of the millennial generation. Data were 
collected from the students using surveys throughout an information search process as part of an 
assigned research project. 

Analysis. Quantitative analysis was carried out on the data, which related to 80 individual 
subjects and evaluation of 758 documents. 

Results. Statistically significant findings suggest that millennial generation Web searchers 
proceed erratically through an information search process, make only a limited attempt to 
evaluate the quality or validity of information gathered, and may perform some level of 
'backfilling' or adding sources to a research project before final submission of the work. 
Conclusions. These findings indicate that the search behaviour of millennial generation 
searchers may be problematic. Existing search models are appropriate; it is the execution of the 
model by the searcher within the context of the search environment that is at issue. 
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Introduction 


The generation born between 1982 and 2000 has been identified as the millennial generation f Strauss and Howe iqqi : 













Howe and Strauss 2000 ). They have come of age in a digital world with ubiquitous information sources. The libraries 
with their strong mediated search support are no longer the primary sources of information for them. Consequently, 
existing information behaviour models may not adequately describe their approach to filling information needs. The 
goal of this research was to explore the information search behaviour of this generation in relation to commonly cited 
information search behaviour models. 

The longitudinal study detailed in this paper evaluated the search behaviour of millennial generation students 
conducting information searches in a naturalistic environment. In the study participants provided detailed information 
on their search choices and judgments of document relevance over a five week period. The findings based on these 
empirical results provide some indication of how members of the millennial generation progress through a search 
process. The subjects in this study proceeded erratically through their search process and did not appear to validate 
sources or information gathered. 

Information search process models provide a framework for the study and evaluation of information searches. Models 
such as Kuhlthau's ( loot ). Ellis's ( iq8q ! and Wilson's ( iqqq ) provide some guidance on the search process but have 
limitations in their application. These models were developed prior to the widespread use of the Internet as an 
information resource. Ellis's subjects were adult professionals, not millennial generation students. Kuhlthau's subjects 
were from a variety of age groups and may have included some young millennial generation subjects, but the research 
was done before the availability of the Internet. 

Research has provided little insight into how information behaviour differs on the Internet, specifically with the 
millennial generation, and whether or not previously identified models are appropriate, millennial generation users are 
more likely to search for information on the Internet using commercial search engines f Abram 2006 : Oblinger 2003). 
Research on how these searches are conducted could inform us on the current information behaviour of the millennial 
generation. Results could provide the basis for improved information search process models which better reflect current 
technology and the generation of information seekers raised with this technology. 

Literature review 

In Generations Strauss and Howe t iQQi l present a broad historical panorama of generations and their influence on 
history. The authors identify the millennial generation as those born in or after 1982. In millennials Rising the authors 
identify the same start date for the generation and infer a cut-off date of 2000. The authors present theories concerning 
the influences of a generational cohort and the environment in which the cohort is raised and comes of age. The 
millennial generation is a group raised during a boom period. This prosperity created significant social and 
technological shifts leading to a distinct cultural identity. 

The millennial generation is only familiar with a world where personal computers and information are easily accessible, 
and has distinct expectations concerning technology, communication, and access to information. Abram (2006) notes 
that millennial generation students expect instant access to information (cell phones, portable computers, Internet 
access), and tend to procrastinate with research and the choices it requires. They tend to be multi-taskers, often doing 
several tasks at once. They exist in a noisy, media-driven world, a condition that may lead to issues with filtering what 
is valid and important to their task. Constant socializing in a connected world leads to persistent distractions from any 
assigned task ( Essinger 2006 ). The prevailing educational view is that students in undergraduate programs have 
incomplete cognitive thinking skills. This creates difficulty in discerning valid information from invalid information. In 
a mediated search environment, a librarian would help the student identify valid information sources and provide 











guidance on how to identify valid and useful information. This mediation is lacking in an environment where the 
student searches the Web unaided f Wicler 2004 ). 

Gross and Latham (2011) conducted an investigation of experiences and perceptions of information with first year 
college students, members of the millennial generation. The researchers indicated that subjects viewed finding 
information as a product, not a process. Subjects in the study indicated that they did not perceive finding information as 
being difficult or requiring any particular skill set. One subject commented that 'the computer does all the work for you 
basically'. Subjects also felt the evaluation of information was subjective, not objective, suggesting a postmodernist view 
of information and knowledge in general. The focus on a search product and dismissal of a process for evaluating and 
verifying content by subjects is reflective of the millennial generation's neoliberal social view C Howe and Strauss 2000 1 
where production and efficiency are important, and what appears to be a consumerist approach to education is a side 
effect of this view. 

In Rowlands et al. (2008), the search behaviour of a subset of the millennial generation is examined in a broad, multi- 
method study. Researchers used a sample of those born after 1993, young members of the millennial generation. They 
noted that searchers of this generation tended to search horizontally rather than vertically, skimming content, viewing 
just one or two pages, and making quick relevance judgments based on this review. This thin, surface-level review of 
content is related to the lack of concern, or inability to discern, the authority of sources and the quality of content, as 
revealed in other studies of members of the millennial generation f Williams and Rowlands 2007 : Hirsh iqqq : Grimes 
and Boening 2001 : Lorenzen 2001 L The familiarity and comfort with the Web as an information search conduit creates 
other problems. The hypertext interface that is the foundation of the Web creates a fragmented view of information. The 
millennial generation is at ease in this environment, often not recognizing incomplete fragments as such. Each fragment 
of information appears to be as valid as any other. In Lorenzen ( 2001 !. subjects were clearly hesitant when asked how 
they evaluate the quality of a Website, some suggesting that a .com domain implied it was a reliable information source, 
and others indicating that a .com site was not a reliable source. Some subjects in the study felt that if a site was indexed 
by Yahoo! it had to be authoritative. 

From a broader perspective, the culture within which the millennial generation has learned cognitive skills may be 
partly to blame for these problems. Philosophers have identified the postmodern condition as a rejection of the 
objective, scientific evaluation of knowledge, and the general acceptance of a an open, subjective, uncritical view of 
knowledge. There is growing concern that members of the millennial generation have adopted this uncritical view of 
information f Harlev et al. 2001 L Millennial generation information searchers appear to be more concerned with the 
time it takes to find and evaluate content than the overall validity or quality of that content, and generally prefer visual 
information to textual f Harlev et al. 2001 : Vondracek 2007 : Wieler 2004 L An additional cultural influence is the 
consumerist society within which the millennial generation has been raised. In Harley et al. (2001), the postmodern 
condition as applied to the dissemination of information is considered a disruptive force in the growth and distribution 
of knowledge in our culture. Consumerism, superficiality and knowledge fragmentation are a result. In the information 
economy, information is just another economic commodity which is consumed at the lowest cost. The cost in this 
context is effort, and when seeking information, the millennial generation individual often perceives the lowest cost as 
the most convenient, readily available information with limited consideration for quality f Buczvnski 200^ : Thompson 
2003.; Young and Von Segern 2001L 


Information search process 

How millennial generation searchers work to fill their information need can be examined within the framework of an 





















information search process model. Kuhlthau ( toot . iqqi ) modelled information seeking as a series of stages. Her model 
was built on personal construct theory, Taylor's (1968) stages of need formation, Belkin et al. 's (1982) anomalous states 
of knowledge, and theories and models of expression and mood. Though developed in 1991, more current research 
continues to validate the basic tenets of the model f Kuhlthau et al. 2008 ). Kuhlthau's model is often interpreted as 
sequential, but the model does specifically allow backtracking and iteration. Ellis f iQ8o . 1 QQ 7 1 identified several search 
patterns used for gathering information to fulfil an information need. His model emphasized the subjective, iterative 
nature of the search process and avoids suggesting there is a consistent sequential process taking place. Wilson f iqoo l 
provided a synthesis and comparison of Ellis's and Kuhlthau's models which maps Ellis's behaviour into Kuhlthau's 
stages. 

Research questions 

In order to better understand the information behaviour of the millennial generation, this research tracked millennial 
generation subjects as they progressed through a five week research project. The goal was to gather information to 
answer the broad question of how information seekers of the millennial generation progress through the process of 
finding information. Specifically, do they proceed through an orderly search process, or is their behaviour somewhat 
erratic and symptomatic of procrastination? Do millennial generation information seekers make some effort to discern 
the quality of documents reviewed on the Web, and if so, are they discerning throughout the search process? In general, 
is the search process used similar to previously identified search process models, or are there indications that their 
search behaviour has changed? These broad questions lead to the following specific research questions. 

1. Do millennial generation information seekers progress through a search process similar to that of Wilson's I iqqq I 
consolidated model, or is their search behaviour different? 

2. Do millennial generation information seekers evaluate the quality of Web resources? Are they discerning about 
quality-related attributes of the sources retrieved from the Web? 

3. How do millennial generation information seekers make use of general information Websites such as Wikipedia? At 
what stage(s) in the search process are these pages used? 

Research design 

The study involved the collection of data based on user searches. Users were millennial generation students in an 
undergraduate business course. Subjects were given an assignment to create a presentation on a computer technology. 
The assignment was structured as a research report and subjects had to complete a bibliography of their sources. Most 
subjects had little knowledge of the research topic they were assigned. Data were collected on the results of each 
subject's information search. All searches were conducted on the Internet. Specifically information was collected on the 
document selected, the relevance judgment for the document (was the document relevant or not), the criteria used to 
judge document relevance, and the subject's stage in an information search process. 

Data were collected using a Website with a modified search engine which executed searches using the Yahoo! search 
engine and then reformatted the results page to provide Web-based data collection for this study. Subjects were able to 
access this search engine interface over the Internet, thus allowing them to work in a naturalistic environment at their 
own pace. Subjects were not required to use the Website for searches, but they were required to use the Website to 
provide data on the results of their searches. Those who did not use the default search engine entered information on 
their sources using survey instruments identical to those used to enter the information from the default search engine. 











Approximately four source documents were entered by each individual using this mechanism. Almost all subjects chose 
to use the default search engine and almost all entries were entered using the survey instruments on those pages. It is 
worth noting that good sources of information could be found on the Web for the computer-related technical topics 
assigned to the subjects. 

Approximately 80 subjects were drawn from a convenience sample of junior and senior undergraduate students at an 
American university. Data were collected in 2007 and subjects were on average 19 through 22 years of age, and were 
therefore born between 1985 and 1988, members of the millennial generation. Subjects signed an informed consent 
form which explained the purpose of the research and that the information they provided would be treated 
anonymously. Subjects were allowed to choose a research topic from a list of topics (see Appendix A). Topics were all of 
the same approximate level of difficulty and were related to course content. Subjects were given several weeks to 
complete their research assignment and were required to complete specific interim assignments during that time 
period. A project abstract was due the first week, a detailed outline was due next, the rough draft of the presentation 
slides due the next week, and the final presentation slides were due the final week. 

Project data were collected anonymously using survey instruments integrated into the Web search engine interface 
(Figure 1). 



Figure 1: Search engine interface 

Subjects examined the documents returned from the search engine and then used the Website to indicate the relevance 
of the documents they examined (relevant, not relevant, partially relevant/not sure about relevance), their current stage 
in an information search process selected from a predetermined list of search stages, and the criteria used to make that 
relevance judgment selected from a predetermined list of relevance criteria (Figures 2 and 3). 


































Figure 2: Search results page 

Note: relevance judgment choices are presented in a drop-down list which presents a mutually exclusive choice of 

relevant, not relevant, and partially relevant or unsure about relevance. 
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Figure 3: Recording the search stage 



Figure 4: Criteria for relevance judgements 





























The Website used for data collection contained detailed instructions on the use of the site and how to record the 
information required. Subjects were given instructions on the use of the Website and the meaning of the terms used on 
the site and help pages were provided to explain the search stage choices, relevance judgment choices, and criteria for 
relevance judgment choices used by the subjects. The Website captured data for each subject using an anonymous user 
ID for search stage choices, the document selected as a uniform resource locator (URL) and the relevance judgment. 

Data collected on the search stage provided an indication of how subjects were progressing through an information 
search process and enabled the evaluation of research question one. The relevance criteria related to quality of the 
document (accuracy, source quality, etc.) provided information for evaluation of research question two, and also 
provided some indication of how and when subjects were evaluating documents, data relevant to question one. Data 
collected on Websites visited (URLs) provided information for research question three. When these Websites were 
being visited and the criteria used during each search stage also provided information about how subjects were 
progressing through the search process, data relevant to research question one. 

Search process, stage in task completion, and relevance criteria choices 

The search stage model used for this research (see Table l) was developed from the phases suggested by Wilson (iqqq) 
which synthesized the information behaviour of Ellis ( 1007 ) with the search process of Kuhlthau f iooi ). The list 
presented to the subjects is shown in Description Displayed to Subject column in Table 1. The term initiation was 
selected because it was considered clearer than beginning or starting. The term browsing was avoided because this term 
is now commonly used for all Web-related searching and would potentially be confusing to subjects. Instead, the term 
exploration was used to describe the process of scanning and gathering information. The remaining terms, 
differentiating, extracting and verifying are from Ellis ( 1QQ7 ) and were used in lieu of Kuhlthau's terminology because 
they were considered more precise. The process of ending the search was considered to be implied by the submission of 
the final deliverable and would not have generated data relevant to the research questions; it therefore was not 
presented as a choice to the subjects. 

Table 1 contains the criteria identified by subjects as having some bearing on their relevance decision (relevant, not 
relevant, don't know or partially relevant), the decision whether or not the document (or Website) is relevant to their 
information need. The relevance criteria chosen by subjects were those factors that contributed to their relevance 
decision. Prior research by Barry 1 1004 !. Barry and Schamber f ioo8 L and Cool et al. f 1903 ) identified the criteria listed 
in Table 1. A number of studies have identified these criteria and have provided some confirmation as to their 
consistency across information retrieval tasks I Barrv and Schamber 1008 : Park iqqi : Schamber iqqi : Schamber and 
Bateman iqq 6 : Xu and Chen 2006 !. A subset of these criteria was presented to subjects not as specific criteria but using 
the contents of the Description displayed to the subject column in Table 3 (see Appendix C). Subjects were allowed to 
choose one or more criteria which they felt contributed to their relevance decision. 

The criteria selected for this research are a subset of those identified in Table 1. The reasons for these selections are as 
follows. The Source column in Table 1 identifies the source of the relevance criterion: Barry f lood ), or Barry and 
Schamber ( 1008 ) , or Cool et al. ( loot ). To reduce the potential for confusion and survey exhaustion on the part of the 
subject, the number of relevance criteria presented was limited to fifteen. In addition to some criteria from the Cool 
study, criteria identified in the Barry f lood ), and Barry and Schamber f ioo8 ) studies were used. Excluded were those 
criteria that were specific only to Schamber’s f iooi ) earlier study and related to document qualities specific to her topic 
(weather reports) and did not apply to the topics used in this study. 






















Criteria 

Type 

Source* 

Used 

Description displayed for subject 

Depth, scope or 
specificity 

Document 

Barry, 

Cool 

Yes 

Document contains good depth on the topic 

Accuracy or validity 

Document 

Barry, 

Cool 

Yes 

Document appears to be accurate 

Currency (recency) 

Document 

Barry, 

Cool 

Yes 

Information is current, recent, up-to-date 

Tangibility 

Document 

Barry, 

Cool 

Yes 

Information relates to real, tangible issues; not esoteric or theoretical 

Quality of sources 
(source quality) 

Document 

Barry, 

Cool 

Yes 

Source is reputable, trusted, considered expert 

Availability of information 

Situation 

Barry, 

Cool 

Yes 

The extent to which the information is available 

Verification 

Document 

Barry, 

Cool 

Yes 

The information is consistent with the body of knowledge the field; 
the information supports the user's point of view 

Affectiveness 

Document 

Barry, 

Cool 

Yes 

The user's emotional response to the information; pleasure, 
enjoyment, entertainment 

Effectiveness of proposed 
approach 

Document 

Barry 

Yes 

How effective is the approach proposed? 

Consensus within the 
field 

Document 

Barry 

Yes 

How much consensus there is in the field for what is proposed in the 
document 

Time constraints 

Situation 

Barry 

Yes 

How much time is allowed for the task to be completed? 

Background, experience 
or ability to understand 

Situation 

Barry 

Yes 

Expression of concern over the ability to understand a document 
(same as 'understandability') 

Novelty, content novelty 
or source novelty 

Document 

Barry 

Yes 

The source or content of the document is new to the subject. 

Geographic proximity 

Document 

Schamber 

No 

Refers to weather information in a geographic location 

Dynamism 

Document 

Schamber 

No 

Refers to the ability to dynamically manipulate the information in a 
document 

Presentation quality 

Document 

Schamber 

No 

Iindication that the source of the information could be manipulated in 
some way. 

Structure 

Document 

Cool 

Yes 

The structure of the document; how the information is presented and 
organized 

Amount of information 

Document 

Cool 

Yes 

Document provides sufficient information. 

Depth 

Document 

Cool 

Yes 

Document covers the topic in good depth. 

Timeliness (age of 
document) 

Document 

Cool 

Yes 

Is the time frame of the document appropriate; (current where 
recent information is required; written in a certain time period for 
historical significance) 

Understandability 

Document 

Cool 

Yes 

The document is understandable by the subject (ability to 
understand) 

Bias 

Document 

Cool 

Yes 

The Document is written with a particular viewpoint. 

Authority 

Document 

Cool 

Yes 

The author or publication has a good reputation in this field. 


Table 1: Relevance criteria 

* from Barry (1994: 154); Barry and Schamber (1998: 226); and Cool etal. (1993: 3). 


Deliverables required of the subject and the month and day they were due are identified in Table 2. Some feedback was 
provided upon submission of the abstract and detailed outline deliverables to encourage the students to produce a 

































thorough and well-researched presentation. Feedback concerned only the technical aspects of the topic and the scope of 
their final presentation. Feedback did not concern choice of Websites, search stage choices or relevance criteria choices 
made by the subjects. 

The final deliverable was a ten to thirty slide PowerPoint presentation. Subjects were encouraged to go beyond general 
information sites (such as Wikipedia), and to use quality sources such as trade journals whenever possible. (Computer 
trade journal content is widely available on the Web.) Subjects were also encouraged to use between ten and twenty 
sources to prepare their presentation. 


Deliverable 

Description 

Due date 

Abstract 

Several paragraphs explaining what will be covered in the 
presentation. 

2 

November 

Detailed outline 

One or more pages of outline text which explain the topic, and 
subtopics which to be presented and the flow of the presentation 

10 

November 

Rough draft of 

presentation 

slides 

PowerPoint slides containing the rough draft of the presentation 

20 

November 

Final presentation 
slides 

PowerPoint slides containing the final presentation 

7 

December 


Table 2: Deliverables used 


Results 


A total of 80 subjects participated in the study and examined and provided 758 distinct Web page evaluations. Table 5 
identifies the number of documents evaluated by subjects for each project deliverable. This table shows that the number 
of documents evaluated varied significantly for each deliverable (?2 = 98.9, df = 3, p-value <0.001). Subjects evaluated 
a total of 81 documents for the project abstract deliverable, but evaluated 225 documents for the detailed project 
outline. This finding is not unexpected considering the subjects' instructions for the preparation of the report abstract 
were to create a brief explanation of the project and thus did not require a detailed knowledge of the topic. Table 3 also 
shows that document selection was evenly distributed over the duration of the research project. 


Project deliverable 


% of total 

Abstract 

81 

10.69% 

Detailed outline 

225 

29.68% 

Rough draft 

187 

24.67% 

Final presentation 

265 

34.96% 

Total 

758 

100.00 


Table 3: Document count by deliverable due 


Table 4 lists the number of search stages reported by subjects. As this table indicates, only five subjects reported being 
in all the search stages provided as options, and a significant number of subjects (67%) reported being in fewer than 
four (?2 = 9.6, df = 1, p-value <0.001). 


Stages reported 

Users reporting 

























5 

5 

4 

22 

3 

20 

2 

20 

1 

13 

Total 

80 


Table 4: Search stages reported by subjects 


Table 5 and 6 identify the percentage of documents evaluated in each search stage. Table 5 lists percentages within 
search stage, and Table 8 lists percentages within project deliverable. This provides some indication of how subjects 
proceed through the information search process. As subjects evaluated documents, they indicated which search stage 
they were in based on a list of search stage explanations (Table 2). Expectations are that research done for preparation 
of the first deliverable, the project abstract, would be the initiation stage (identified as the 'initial search; start of the 
search process'). As Table 7 indicates, slightly less than half of the subjects indicated that they were in the initiation 
stage while preparing the project abstract. This finding indicates that over half the subjects reporting for the project 
abstract, the first deliverable, reported being in a search stage other than the initiation stage. 


Table 6 provides another perspective on the initiation stage and the relationship between search stage and deliverable. 
As a percentage of documents evaluated within a search stage, during the preparation of the detailed outline, 
approximately 50% of the subjects reported being in the initiation stage, and they reported this after completing the 
project abstract, the first deliverable. Additionally, 13% reported being in the initiation stage during the preparation of 
the rough draft, and 11% reported being in the initiation stage during the preparation of the final presentation. This 
indicates a non-serial progression through the information search process in relation to project deliverables. 


As Table 5 indicates, only 16% of the subjects reported being in the verifying stage while preparing the final 
presentation deliverable. Subjects were more likely to identify extracting as their current search stage (defined as 
‘extracting information to answer the question’) during this time frame. This indicates that these subjects were not 
concluding the search process in a manner consistent with Kuhlthau's information search process which has an 
expectation that the subject will perform some verification or evaluation of the research collected in the final stage. 


Search stage 

Abstract 

Detailed outline 

Rough draft 

Final presentation 

Initiation 

47.08% 

25.32% 

16.25% 

8.89% 

Exploration 

22.81% 

25.67% 

9.96% 

12.72% 

Differentiating 

7.48% 

10.57% 

2.85% 

9.22% 

Extracting 

20.62% 

35.69% 

65.84% 

53.18% 

Verifying 

2.01% 

2.75% 

5.10% 

15.99% 

Total 

100.00% 

100.00% 

100.00% 

100.00% 


Table 5: Percent of documents evaluated by search stage within deliverable 


Search stage 

Abstract 

Detailed outline 

Rough draft 

Final presentation 

Total 

Initiation 

25.75% 

49.70% 

13.67% 

10.88% 

100.00% 




































Exploration 

14.37% 

58.05% 

9.66% 

17.93% 

100.00% 

Differentiating 

10.62% 

53.89% 

6.22% 

29.27% 

100.00% 

Extracting 

5.59% 

34.72% 

27.45% 

32.25% 

100.00% 

Verifying 

3.62% 

17.76% 

14.14% 

64.47% 

100.00% 


Table 6: Percentage of documents evaluated by deliverable within search stage 


Evaluation of general information sites 

The selection of general information sites (defined in this study as the Wikipedia site) varied significantly in relation to 
project deliverable due (?2 = 12, n=8o, df = 3, P <0.01) as shown in Table 7. This site selection was approximately 10% 
of the total documents chosen overall (80 out of 758). Subjects were instructed to search beyond basic informational 
sites, and there were numerous sites available for their assigned topics. As Table 7 indicates, these sites were used 
consistently during the preparation of the abstract and rough draft, ranging between approximately 6% and 10% of the 
total sites chosen. There was a significant increase in the usage of these sites during the preparation of the final 
presentation. Table 8 provides additional insights into the use of these sites, showing a statistically significant 
relationship between the search stage and the selection of Wikipedia Web sites (?2 = 15.7, df = 3, p-value < 0.001). 
These results indicate that Wikipedia sites represent approximately 40 % of the of the sites evaluated when subjects 
reported being in the 'extracting' stage (identified as "extracting information to answer the question"). 


Deliverable due 

Count 

Percentages* 

Abstract 

8 

9.30% 

Detailed outline 

29 

9.90% 

Rough draft 

15 

5.84% 

Final presentation 

28 

14.97% 

Total 

80 



Table 7: Wikipedia pages reviewed by deliverable due 

* Percentage of Wikipedia pages within all pages evaluated for the 
deliverable due 


Stage code 

Count 

Percentages 

Initiation 

20 

25.97% 

Exploration 

17 

22.08% 

Differentiating 

6 

7.79% 

Extracting 

31 

40.26% 

Verifying 

3 

3.90% 

Total* 

77 

100.00% 


Table 8: Wikipedia pages reviewed by search stage 

* three subjects did not report a search stage 

Evaluation of criteria used to judge relevance 


An examination of the criteria used by subjects to determine relevance provides some indication of their reasons for 






























selecting or rejecting documents as they progressed through an information search process. Since there were a large 
number of criteria examined by the subjects, Table n is simplified to identify those criteria which changed most in 
terms of their selection as the subject moved through the search process. As this table indicates, subjects were more 
likely to select amount of information and depth as criteria in the earlier search stages (initiation and differentiating) 
than in later stages (initiation and verifying). The criteria of novelty of sources, the structure of the document, and time 
constraints of the subject was selected more in later stages. This may indicate that in later search stages the subjects are 
looking for new sources of information and possibly documents of different structure, providing a different approach 
and new information for their subject area. The criterion of time constraints was also more likely to be selected later in 
the process as deadlines loom. The criteria of and depth were selected less later in the search process, when other 
criteria such as authority and structure were more likely to be selected. Though authority was selected more in later 
stages, the criterion of source quality was selected less, suggesting that based on the subjects' selections, subjects may 
not apply equal weight to authority and source quality as criteria. 


Relevance criteria 

Initiation 

Differentiating 

Extracting 

Verifying 

Affectiveness 

0.57% 

4.37% 

2.77% 

3.52% 

Amount of information 

8.92% 

12.23% 

10.77% 

7.42% 

Authority 

1.90% 

3.49% 

2.65% 

3.91% 

Bias 

1.90% 

1.75% 

1.94% 

3.13% 

Depth 

8.92% 

12.23% 

9.03% 

7.42% 

Novelty 

1.14% 

3.93% 

2.06% 

2.73% 

Recency 

8.35% 

7.86% 

6.97% 

6.25% 

Source quality 

4.93% 

3.49% 

4.06% 

3.13% 

Structure 

6.26% 

5.68% 

7.55% 

8.20% 

Time constraints 

0.57% 

1.31% 

1.29% 

2.34% 


Table 9: Stage/criteria percentages (reported as a percentage of the total choices within a 

search stage) 


An examination of results grouped by deliverable required provides a slightly different perspective on the progress of 
the subjects, as shown in Table to. This table examines criteria selected based on the date the subject made the 
evaluation. The date was then correlated with the deliverable due in that time frame. Based on this evaluation, subjects 
working on the final presentation, subjects who were approaching the end of their research effort, were more likely to 
select amount ofinformation and understandability as criteria in their document choice. The accuracy, source quality, 
depth and breadth, however, were less likely to be selected by these subjects. 


Criteria code 

Rough draft 

Final Presentation 

Accuracy 

14.66% 

11.71% 

Affectiveness 

3.02% 

5.43% 

Amount of information 

12.07% 

14.00% 

Authority 

3.23% 

4.29% 

Bias 

3.23% 

2.14% 

Breadth 

7.76% 

6.71% 

Depth 

13.36% 

10.29% 

































Discussion 


Novelty 

1.94% 

3.14% 

Recency 

8.84% 

9.14% 

Source quality 

5.17% 

4.00% 

Structure 

8.41% 

9.14% 

Time constraints 

2.80% 

2.43% 

Understandability 

15.52% 

17.57% 


Table 10: Comparison of criteria codes for rough draft and final 

presentation 


In evaluating results in reference to research question one, these empirical results suggest that millennial generation 
subjects do not proceed in an orderly fashion through an information search process. Though models such as 
Kuhlthau's ( 1993 ) did not predict a strictly linear progression through the information search process, it was implied 
that subjects would move through each of the various steps in the process. Subjects in this sample gathered documents 
consistently over the course of the assignment (see Table 3). Subjects should have acquired good knowledge of the topic 
by the time they completed their rough draft, but with this sample approximately 35% of the documents used were 
retrieved after the rough draft was completed, suggesting procrastination or backfilling (adding sources late in the 
research process as they finalize their report) on the part of subjects. A significant portion of subjects (67 %) reported 
being in fewer than four of the search stages studied (Table 4) also providing some indication that subjects may not be 
moving through a search process in an manner consistent with the search stages used in this study. 

If subjects had made an orderly, pragmatic progression through an information search process, it would be expected 
that they report they were ‘verifying information gathered previously during the preparation of the final project 
deliverable. This search progression would be based on the expectation that subjects had gathered research previously, 
and were now examining and verifying that research as part of completing the final deliverable. The empirical evidence 
gathered with this subject pool suggests otherwise. The majority of subjects (53%) evaluating documents for the final 
presentation identified themselves as being in the extracting stage, described as ‘extracting information to answer the 
question’. Only 16% of subjects preparing the final deliverable identified themselves as being in the verifying stage. A 
closer examination of the verifying stage indicates that the selection of this stage as a percentage of the stages reported 
for a particular deliverable never exceeded 15% of the total (see Table 5), an indication that subjects were not ‘verifying 
information that has been gathered previously’. Most of the documents evaluated by the subjects (approximately 70%) 
were evaluated in the final two stages, the latter portion of the project's duration. This amount demonstrates a 
statistically significant correlation with search stage (?2 = 28.1214, df = 1, p-value <0.001) suggesting that subjects may 
be procrastinating or backfilling. 

Over half the subjects reporting for the first deliverable reported being in a stage other than the initiation stage. This 
finding could be an indication that in this time frame, subjects found several documents and reported being in the 
initiation stage, and then after absorbing those initial set of documents, considered themselves out of the initiation 
stage and in some other stage. Subjects may have performed some significant portion of their research in this early 
stage, quickly moving beyond the initiation stage. 

In reference to research question two, examination of the results concerning subjects' evaluation of the quality of Web 
resources provides some indication that subjects were not concerned with the quality, validity, or authority of the 
documents selected. Subjects selected the categories of source quality and authority (of sources) at a rate of between 











55% or less than that of amount of information and depth (see Table 9) suggesting that these criteria were not as 
important to subjects as simply retrieving some quantity of information on their topic. Categories such as structure, 
amount of information, recency (currency) and depth were consistently selected more often by subjects, suggesting 
these criteria were more important to the subjects than source quality or the . 

Subjects did not select verifying (presented as 'verifying information that has been gathered previously') for any 
particular deliverable at a rate higher than 16% (Table 5), suggesting that subjects generally found some other search 
stage category provided a better description of their behaviour. Based on these results, it appears subjects did not 
attempt to verify their sources, and it can be assumed they considered the information valid or were not concerned 
about the validity of the source. 

In reference to research question three, subjects did appear to make use of general information sites such as Wikipedia, 
but they did not select these sites consistently throughout the search process. The instructions presented to subjects on 
the use of general information sites is obviously an intervening factor to be considered in the evaluation of this statistic. 
However, the timing of the selection and use of these sites by the subjects suggests that they were likely to select these 
sites in the extracting stage. Also, the selection of Wikipedia for approximately 15% of sites for the final deliverable 
suggests that for some subjects, these sites were useful in the later stages of the research project. The difference between 
the selection of these sites in early stages versus late stages, however, is not statistically significant. For this sample, 
subjects appeared just as likely to use these sites in early search stages (initiation, exploration) as in later search stages 
(differentiating, extracting, verifying). 

Limitations 

The subjects in this study were from a university in the USA. The articles referenced in the literature review also cited 
studies performed on subjects in schools in the USA. It is not clear that search behaviour, or more broadly, the 
behaviour of millennial generation subjects, will be the same in other cultures. Subjects were also undergraduate 
students with a mixed scholastic background (below average, average, above average). Graduate students or 
professionals may perform differently. 

The task assigned students provided specific instructions to encourage students to perform research. Students were told 
that they were being graded partially on the quality of their research, and this meant that more quality sources cited 
would result in a higher grade. Specific instructions given to the subjects also addressed limiting the use of general 
information sites such as Wikipedia. These intervening factors may have encouraged students to evaluate more 
documents from quality sites and perform additional research during the later stages of the project. In the absence of 
these instructions, it is possible Wikipedia sites may have represented a larger portion of the sites selected and 
referenced. 

This study was designed to allow the subjects to work in an environment outside of a lab at their own pace and at their 
own convenience. The intention was to create an environment where the subject would feel comfortable and behave in a 
typical manner and provide meaningful results on the common information behaviour of the millennial generation. This 
meant that the subjects themselves reported their search stage and criteria and relevance as opposed to the method in 
many information behaviour studies where researchers provide that evaluation. Though subjects were tutored in the 
use of the system and provided with explanations for what was being reported, and though help prompts and help pages 
were available to subjects, there is the possibility that some subjects may have misinterpreted these instructions and 
that these prompts and requirements on reporting may have led to some non-typical behaviour on the part of subjects. 


The task assigned to the students involved a subject area that was suitable for Web-based research. This assignment 
may have encouraged fragmented searches and retrieval since information for many of the assigned topics could be 
retrieved from multiple sources available on the Web. Other assignments, for example writing a paper on Western 
philosophy for a humanities course, might involve more focused searches with fewer sources. 

The research methods employed allowed subjects to select topicality as a criterion, but did not specifically analyse 
topicality in relation to the selection of other criteria. This approach was based on the assumption that the document 
first needed to be on topic before other criteria would be considered f Wang and Sorgel iqq8 : Crystal and Greenberg 
2006). This research focused on the use of more specific relevance criteria. 

Additional influences on relevance decisions are known to be user's background or knowledge of the subject domain, 
and search task. The convenience sample for this research was drawn from a pool of undergraduate students who are 
business majors in a business school at an American university. All students were taking the same course and were 
given the same assignment, thus all had the same work task. These influences are mitigated in this study by drawing 
from a subject pool whose members have similar backgrounds, experiences, and domain knowledge. Though this aspect 
of the design of these studies attempted to control for variations in domain knowledge, variations in knowledge may 
exist within the subject pool. The choice of this convenience sample also limits the generalisability of the results. 

Though subjects could use any search engine to gather sources, the Yahoo! search engine was used by default since it 
was easier to integrate search results into the Web-based data collection instruments used in this study. Almost all 
subjects chose to use the default search engine. The search engine itself and the search results were not being examined 
in this study, but use of a different search engine as the default (for example, Google) may have produced different 
results. 

Conclusion 

These findings suggest that the execution of searches by millennial generation searchers may be problematic. Existing 
search models are appropriate; it is the execution of the model by the searcher within the context of the search 
environment that is at issue, millennial generation searchers have come of age with different information gathering 
resources than their predecessors. The filter of the librarian-mediator relationship is gone and replaced by a profusion 
of fragmented and sometimes dubious information sources. It is within this environment that they search and gather 
the information they need. The results presented here provide some indication that members of the millennial 
generation do not consider verification of Internet sources important, indicating a non-critical view of information 
found on the Internet. This is consistent with results by Gross and Latham ( 2011 ) who report that subjects in their 
study indicated that information quality was not a concern and express surprise and incredulity that there may be an 
objective measure of information evaluation. 

These findings point towards an information literacy problem that may be based more on a perception rather than lack 
of information searching skills. The millennial generation has come of age with technology that provides access to a vast 
variety of information sources. They constantly make choices concerning focus. They clearly know how to filter, the 
problem is what they choose to filter when evaluating information. 

As Maybee ( 2006 ) indicated, learning involves a relationship between the learner and the subject matter. Changing the 
relationship involves changing the learner's perceptions. Subjects identified in Gross and Latham ( 2011 ) were not 
concerned with the quality of information gathered, consistent with the behaviour of the subjects in the study reported 
here who rarely reported validating information sources (identified as source quality). Gross and Latham further 








indicate that subjects did not perceive the existence of an objective standard for evaluating information (see Gross and 
Latham 2011 : 173). Finding and evaluating information is subjective, reflecting a distinctly postmodernist view of 
information and knowledge. 

It is this core perception of information quality as a subjective concept, and perhaps even a broader uncritical view of 
information in general, that must be addressed. Further research should explicate this view and perhaps the broader 
worldview of subjectivity versus objectivity amongst the millennial generation. A better understanding of why members 
of this generation resist using objective information evaluation standards may provide strategies for addressing this 
perception and thus improving the search skills of the members of the millennial generation. 
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Appendix 

Research topics assigned to subjects 

• Computer security: making computer technology accessible and secure 

• Computer security: making desktop systems secure 

• Computer security: preventing computer fraud 

• E-Commerce: after the internet bubble 

• E-Commerce: how to put your company on the Web 

• Internet business models 

• ERP systems: the future 

• Customer resource management (CRM) systems: current status 

• Does IT matter: what role will it take in the future? 
















• New technologies: can Linux be mainstream ? 

• New technologies: the future of WiFi 

• Microsoft: dealing with the 500 pound gorilla 

• Ethics and the information age: is it really stealing if it's digital ? 

• Distributed computing 

• Grid computing 

• Group collaboration with computers 

• Computer aided design (CAD) systems 

• Supply chain management with computers 

• Privacy and computers 

• Decision support systems 

• Implementing enterprise resource planning (ERP) systems 

• Alternatives to ERP systems 

• The current state of artificial intelligence and expert systems 

• Systems design and development 

• Enterprise portals and application integration 

• Open source software on the desktop: current status 

• ERP: implementation issues 
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